The present invention relates to the processing of digital video signals, and more particularly to the multiplexing of digital video programs in an adaptive bit rate environment for transmission over a fixed-bit-rate transmission channel.
Statistical Multiplexing:
Today video service providers, such as cable television operators, direct broadcast satellite companies, and over-the-air digital television broadcasters, transmit digitally compressed video either in a Constant Bit Rate (CBR) form or in a Variable Bit Rate (VBR) form using the MPEG-2 Transport Stream protocol.
If CBR compression and transmission is used, as illustrated in FIG. 1, then a video program is compressed such that the bit rate (in bits per second) of the compressed video program is constant over time throughout the length of the video program. When sending multiple CBR-compressed video programs over a transmission channel, the video service provider sends as many video programs (e.g., programs 1-4) over the channel as possible, such that the sum of all the bit rates of the CBR-compressed video programs is less than the bandwidth of the channel measured in bits per second. CBR compression and transmission is easy to manage, but it is not the most efficient means of sending video over a fixed-rate transmission channel if the goal is to maximize the picture quality of all of the transmitted video programs that are sent over the transmission channel. With CBR compression, while the bit rate is constant, the rendered picture quality of the video program can vary over time.
An alternative method of compression and transmission is to employ VBR compression and statistical multiplexing for transmission. Today video service providers almost universally employ this method of transmission in order to maximize the video quality they offer their subscribers across their multichannel video service.
With VBR compression, nearly constant picture quality over time for a video program can be achieved while letting the bit rate of the compressed video program vary over time. The bit rate varies over time because easy scenes (such as low-detail or slow action scenes) in the video program require less bits to compress to a pre-determined target picture quality while hard scenes (such as high-detail and fast action scenes) take more bits to compress to a pre-determined target picture quality. These “peaks” and “valleys” in bit rate fluctuate over time depending upon the content of the video program. Note that for a CBR-compressed video program, its constant bit rate equals its average bit rate. When compressed at an average bit rate equal to its CBR-compressed version, the VBR-compressed video program achieves higher picture quality, because VBR compression allows the bit rate to peak when needed in order to render hard scenes at the target picture quality, while expending less bits for easy scenes so as not to use more bits than necessary to render the target picture quality.
Also, the location of the “peaks” and “valleys” in the bit rate of one video program are typically uncorrelated with those of another video program, and thus, generally the “peaks” and “valleys” do not align in time from one video program to another. Therefore, the idea of statistical multiplexing is to control where the “peaks” and “valleys” of multiple VBR programs occur and then to combine them into a fixed rate channel in such a way that the “valleys” of some video programs will “absorb” the “peaks” of other video programs, and hence, deliver higher picture quality for all the programs, as compared with using CBR compression, where the constant bit rate is equal to the average bit rate of each of the VBR-compressed versions.
The implementation of statistical multiplexing must make sure, however, to fit all the VBR-compressed video programs into the fixed rate channel since there is no guarantee that all the “valleys” will absorb all the “peaks” perfectly. There will be instances where the aggregate sum of all the VBR-compressed video programs that are sent in a fixed-rate channel want to exceed the fixed rate of the channel. Hence, an implementation of statistical multiplexing needs some way of reducing the bandwidth at any instance in time to make sure that the aggregate sum is less than or equal to the fixed rate of the channel.
Today, video service providers statistically multiplex compressed video programs that they receive from various sources using a statistical multiplexer as in FIG. 2. The output of the statistical multiplexer is a statistical multiplex of video programs which is delivered over the “last mile” of a video service provider's distribution network to the receiving and decoding device (or client) in the home. In implementing statistical multiplexing, video service providers employ Transrating or Transcoding to reduce the bandwidth of a video program at any instance in time.
In statistical multiplexing:                a. Various sources (e.g. sources 1-4) of compressed video (for example, satellite receivers, over-the-air receivers, video encoders that compress baseband video, and video servers) stream or push CBR-compressed and/or VBR-compressed video programs to the statistical multiplexer 10;        b. The pushed video programs are in the form of transport streams. An example of such a transport stream is the MPEG-2 transport stream described in the MPEG-2 Systems specification ISO/IEC 13818. Transport streams, among other things, contain strict timing information and must be delivered or streamed to the multiplexer 10 in strictly timed packets of data;        c. Because the video programs are pushed to the statistical multiplexer 10, the statistical multiplexer 10 is a slave to the sources of video and takes whatever the source provides to it;        d. Transrating can change the compressed bit rate of a video program by various methods, depending upon the manufacturer of the product. It can employ such techniques as changing the quantization level of the compressed video program. Transrating can not only lower the bit rate of a VBR-compressed video program at any instant in time, but can also convert CBR-compressed video programs to VBR-compressed versions;        e. Transcoding involves decoding the video back to its uncompressed format (or ingesting it in this format) and then re-encoding it while controlling the output bitrate under the control of the statistical multiplexer 10;        f. When performing statistical multiplexing, multiple streams (for example, N streams) are streamed or pushed from their sources to the statistical multiplexer 10. The statistical multiplexer 10 combines the streams together and, with the aid of Transrating or Transcoding, either 1) reduces the bit rate of one or more VBR-compressed streams, when needed, to fit all the streams into the fixed-rate output channel or 2) converts the CBR streams into VBR streams and makes sure that all the streams fit within the fixed-rate output channel 12.        
For example, one could send ten 4 Mbit/sec CBR-compressed video programs in a 40 Mbit/sec channel. Alternatively, one could send ten VBR-compressed video programs over a 4 Mbit/sec channel where the average bit rate of each VBR-compressed video program is 4 Mbits/sec. The result is that the picture quality of each of the VBR-compressed video programs will have a more constant and higher picture quality than the CBR-compressed video programs. The employment of statistical multiplexing is desirable for video service providers who want to deliver the highest picture quality in a given amount of bandwidth.
Adaptive Video Delivery:
Adaptive video delivery is a recent technique popularized by technology providers such as Apple, Microsoft and Adobe and employed by Internet video service providers (such as abc.com, Netflix, Fancast.com, etc.) to enable the delivery of uninterrupted video (i.e., without glitches, stops and starts, and pauses) over unmanaged networks, such as those provided by cable operators and telephone companies for normal high-speed data service, which are what people connect over today when watching video over the Internet.
Adaptive video delivery is different from traditional video streaming and transmission techniques used today. Today's video streaming over cable, satellite, over-the-air television, and DVDs utilize a continuous string of data called a Transport stream (such as the MPEG-2 Transport Stream protocol) that contains the compressed video information for the entire video program, as well as timing information used for decoding and the maintenance of strict timing of the delivery of the data to the client or decoder, in one long, continuous file. This transport stream is pushed from the source to the client with the strict timing of the delivery of the data maintained along the transmission path.
In adaptive video delivery:                a. As illustrated in FIG. 3, several versions (e.g., A-E) of a video program 18 are created where each version is the result of compressing the video program 18 at each of several pre-determined constant bit rates. For example a video program can be compressed at 500 Kbits/sec, 750 Kbits/sec, 1 Mbit/sec, 1.25 Mbits/sec, etc. creating a version for each bit rate. Each version A-E is a bit-rate profile of the original video program.        b. CBR compression is used today for each of the different versions or profiles, but VBR compression could also be used.        c. Each version or profile is divided into smaller files, called fragments or chunks 20, where each fragment 20 corresponds to a fixed amount of playback time, for example 2 seconds, as shown in FIG. 3.        d. As shown in FIG. 4, each profile A-E, in its fragmented-file form, is stored on a video server 22 and grouped together with the other profiles for the video program 18.        e. When a client 24 requests a video program 18, instead of the source pushing the entire transport stream of the video program 18 to the client, the client issues a request to the video server for a first fragment 20 of a specific profile or its corresponding bit-rate. In other words, the client 24 pulls the video packets from 1 the source (server 22) one fragment 20 at a time.        f. Once it receives and decodes this first fragment, the client 24 then asks for the next fragment 20 in time but it can ask for this next fragment 20 from any of the bit rate profiles A-E. For example, after receiving the first fragment 20 from the 1 Mbit/sec profile, the client 24 could ask for the second fragment 20 in time from the 750 2 Kbit/sec profile.        g. What determines the profile from which the client 24 should request the next fragment 20 is the local condition of the client 24, which can be things like its computational load or, most commonly, the bandwidth given by the network to the client 24 on the previous request. The bandwidth given by the network at any instance in time can vary over time because of network conditions such as congestion. Therefore, the idea is to match the bit-rate profile of the video program 18 at any 2-second instant in time to the bandwidth given to the client 24 at that instant in time, thus providing glitch-free and pause-free viewing of the video program 18. The client 24 has a narrow view of the bandwidth given to it based upon the how long it takes a fragment to download. For example, if a client 24 requests a fragment 20 from a particular profile A-E and it takes longer than the playback time of the fragment 20 (2 seconds for example) to download, then on the next request, the client 24 will ask for a fragment 20 from then a lower bit rate profile A-E. The opposite will also happen, and if it takes the same amount of time as the playback time of the fragment 20 to download, then the client 24 will request its next fragment 20 from the profile it requested its previous fragment from.        h. This process of subsequent requesting for the next fragment 20 in time from any of the profiles continues until the video program 18 is fully viewed or until the client 24 stops requesting.        i. It should be noted that unlike streaming, in adaptive video delivery the server holds no state information for the client-server pair, while the client holds all the state information for the client-server pair (e.g. video is in a ‘play’ state or in a ‘paused’ state).        
Unlike statistical multiplexing of video today, where the goal is to maximize the video picture quality across multiple video programs that are pushed to the statistical multiplexer and delivered simultaneously in a fixed rate channel, the goal of adaptive video delivery is to deliver a glitch free viewing experience over a channel of varying bandwidth for a single client and video program.
Traditional video service providers such as cable companies are looking at using adaptive streaming techniques to deliver video on their networks instead of using traditional MPEG-2 transport streaming.
With today's transport streaming techniques such as MPEG-2 transport streaming, video service providers are able to maintain a high quality viewing experience of the video programs. This is due to management of the bandwidth assigned to any video program at any given instant in time using, for example, statistical multiplexing during the transrating or encoding process. Such management maximizes the video quality across all of the video programs in a fixed-bandwidth transmission channel and provides a seamless video experience for the viewer.
On the other hand, adaptive video delivery allows video service providers to take advantage of standard off the shelf data servers, open Internet-Protocol (IP) solutions, a common content management system across multiple video delivery platforms, and the faster innovation cycles of the Internet. However, adaptive video delivery solutions are only geared toward managing the delivery of a single program to a single user without knowledge of the bandwidth consumed by or delivery decisions made by other users in the same transmission channel. Also, these decisions are not based upon knowledge that there are multiple video programs in the transmission channel, knowledge of the bandwidth requested by all users, knowledge of the instantaneous quality level of each video program being requested, nor the target goal to maximize picture quality across multiple video programs simultaneously, for example.
As such, there is a need to apply multiplexing techniques to adaptive video delivery so that video service providers can not only take advantage of the benefits of adaptive video delivery techniques but also be able to maximize and manage picture quality across multiple video programs when delivering multiple video programs within a fixed-bit-rate transmission channel. That is, the multiplexing techniques make use of the aforementioned knowledge in order to optimize the delivery of the group of video stream over the delivery channel, rather than optimizing the delivery of just one video stream.
The present invention satisfies this need and provides the foregoing and other advantages.